97 research outputs found
Use of static surrogates in hyperparameter optimization
Optimizing the hyperparameters and architecture of a neural network is a long
yet necessary phase in the development of any new application. This consuming
process can benefit from the elaboration of strategies designed to quickly
discard low quality configurations and focus on more promising candidates. This
work aims at enhancing HyperNOMAD, a library that adapts a direct search
derivative-free optimization algorithm to tune both the architecture and the
training of a neural network simultaneously, by targeting two keys steps of its
execution and exploiting cheap approximations in the form of static surrogates
to trigger the early stopping of the evaluation of a configuration and the
ranking of pools of candidates. These additions to HyperNOMAD are shown to
improve on its resources consumption without harming the quality of the
proposed solutions.Comment: http://www.optimization-online.org/DB_HTML/2021/03/8296.htm
Extensions Ă l'algorithme de recherche directe mads pour l'optimisation non lisse
Revue de la littérature sur les méthodes de recherche directe pour l'optimisation non lisse -- Démarche et organisation de la thèse -- Nonsmooth optimization through mesh adaptive direct search and variable neighborhood search -- Parallel space decomposition of the mesh adaptive direct search algorithm -- Orthomads : a deterministic mads instance with orthogonal directions
Quantifying uncertainty with ensembles of surrogates for blackbox optimization
This work is in the context of blackbox optimization where the functions
defining the problem are expensive to evaluate and where no derivatives are
available. A tried and tested technique is to build surrogates of the objective
and the constraints in order to conduct the optimization at a cheaper
computational cost. This work proposes different uncertainty measures when
using ensembles of surrogates. The resulting combination of an ensemble of
surrogates with our measures behaves as a stochastic model and allows the use
of efficient Bayesian optimization tools. The method is incorporated in the
search step of the mesh adaptive direct search (MADS) algorithm to improve the
exploration of the search space. Computational experiments are conducted on
seven analytical problems, two multi-disciplinary optimization problems and two
simulation problems. The results show that the proposed approach solves
expensive simulation-based problems at a greater precision and with a lower
computational effort than stochastic models.Comment: 36 pages, 11 figures, submitte
Constrained stochastic blackbox optimization using a progressive barrier and probabilistic estimates
This work introduces the StoMADS-PB algorithm for constrained stochastic
blackbox optimization, which is an extension of the mesh adaptive direct-search
(MADS) method originally developed for deterministic blackbox optimization
under general constraints. The values of the objective and constraint functions
are provided by a noisy blackbox, i.e., they can only be computed with random
noise whose distribution is unknown. As in MADS, constraint violations are
aggregated into a single constraint violation function. Since all functions
values are numerically unavailable, StoMADS-PB uses estimates and introduces
so-called probabilistic bounds for the violation. Such estimates and bounds
obtained from stochastic observations are required to be accurate and reliable
with high but fixed probabilities. The proposed method, which allows
intermediate infeasible iterates, accepts new points using sufficient decrease
conditions and imposing a threshold on the probabilistic bounds. Using Clarke
nonsmooth calculus and martingale theory, Clarke stationarity convergence
results for the objective and the violation function are derived with
probability one
A general mathematical framework for constrained mixed-variable blackbox optimization problems with meta and categorical variables
A mathematical framework for modelling constrained mixed-variable
optimization problems is presented in a blackbox optimization context. The
framework introduces a new notation and allows solution strategies. The
notation framework allows meta and categorical variables to be explicitly and
efficiently modelled, which facilitates the solution of such problems. The new
term meta variables is used to describe variables that influence which
variables are acting or nonacting: meta variables may affect the number of
variables and constraints. The flexibility of the solution strategies supports
the main blackbox mixed-variable optimization approaches: direct search methods
and surrogate-based methods (Bayesian optimization). The notation system and
solution strategies are illustrated through an example of a hyperparameter
optimization problem from the machine learning community
Parallel Space Decomposition of the Mesh Adaptive Direct Search Algorithm
This paper describes a Parallel Space Decomposition (PSD) technique for the Mesh Adaptive Direct Search (MADS) algorithm. MADS extends Generalized Pattern Search for constrained nonsmooth optimization problems. The objective here is to solve larger problems more efficiently. The new method (PSD-MADS) is an asynchronous parallel algorithm in which the processes solve problems over subsets of variables. The convergence analysis based on the Clarke calculus is essentially the same as for the MADS algorithm. A practical implementation is described and some numerical results on problems with up to 500 variables illustrate advantages and limitations of PSD-MADS
- …